Remark: yeah, I'm not sure how to detect this properly without making some convoluted logic, even though I imagine this would be one of the more common mistakes to make. At least this does error, maybe we could provide some kind of contextual HELP like if we see --- (3+ starting dashes)

error: expected item, found `-`
  --> $DIR/frontmatter-after-tokens.rs:3:1
   |
LL | ---
   | ^ expected item
   |
   = note: for a full list of items that can appear in modules, see <https://doc.rust-lang.org/reference/items.html>
   = help: if you meant to write a frontmatter, the frontmatter must come after an optional shebang but before any regular source code

(With some better wording, I find it tricky to explain)

Words are hard 🙂

If the file doesn't start with a shebang line, maybe we could just say something like

   = help: if you meant to write a frontmatter, the frontmatter must appear first in the file

Shebangs are not super commonly used so if we can leave that detail out, it would simplify the error message.

I'm pretty sure this can be achieved not at the lexer/parser level but at the tokens to AST level. That code would not reside in places this PR touches so we can do this as a separate followup! :)

Words are hard 🙂

If the file doesn't start with a shebang line, maybe we could just say something like

   = help: if you meant to write a frontmatter, the frontmatter must appear first in the file

Shebangs are not super commonly used so if we can leave that detail out, it would simplify the error message.

Now that we have an implementation using the lexer, instead of stripping (#137193), what are your thoughts on the trade offs?

The cases I can think of

• General code design. I can't speak to this
• Other tools (r-a, rustfmt, etc). I suspect this will need slightly more work for them to passively support the new syntax. For richer features (processing the content, mutating), either they'll still need their own parser or the token's capabilities will need to be expanded which is where we were at with Add unstable frontmatter support #137193
• Other uses for the token existing? I can't think of any but maybe you have some in mind.
• Diagnostics: Provide helpful diagnostics for shebang lookalikes #137619 shows that stripping can still have decent diagnostics. I can't speak to how bad it would be to implement each side

#137619 still does error reporting in the parsing step, which still suggests we can't report errors in the lexer level. As far as my knowledge of the compiler goes, parsing it as a token and reporting it when the token gets "cooked" in the parsing step is pretty much the best way. That is how the implementation for c"str" literals worked and it was stabilized with that implementation.

nit: Rather than searching for use then //! then #![, it might be better to find the first occurrence of any of those. Might also be good to search for "\nuse " as you could imagine a frontmatter of Cargo.toml using a user_auth = 1.0 dep or something.

Not a big deal though and can certainly be changed later 🙂

I've changed it to "\nuse " but left the second part of that. I've noted down the other part of this suggestion in my todo list, because I think it would require pulling in aho-corasick, which is probably more suitable as a followup.